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ABSTRACT 


Post-Traumatic  Stress  Disorder  (PTSD),  major  depression,  anxiety  disorders,  and  other 
post-deployment  adjustment  difficulties  affect  a  significant  number  of  veterans  returning 
from  Operation  Iraq  Freedom  and  Operation  Enduring  Freedom.  To  contend  with  this 
new  influx  of  veterans  suffering  from  the  psychological  aftermath  of  military  combat,  the 
VA  has  been  proactive,  including  commissioning  this  study  of  their  PTSD  health-care 
delivery  system.  Its  objective  is  to  provide  the  best  care,  in  the  most  efficient  manner 
possible  to  as  many  affected  veterans  as  possible. 

This  study  examines  PTSD  health-care  delivery  from  a  systems  engineering 
perspective.  It  employs  state-of-the-art  tools  such  as:  ExtendSim  modeling  and 
simulation  software,  and  JMP  analysis  software. 

The  resulting  models  produce  a  set  of  eight  optimized  system  factors,  which 
maximize  the  desirability  of  four  system  performance  measures  that  define  the  efficiency, 
capacity,  and  timeliness  of  the  system.  We  argue  that  these  models  can  and  should  be 
used  as  a  platform  for  future  work  in  this  area  of  study. 
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EXECUTIVE  SUMMARY 


Post-Traumatic  Stress  Disorder  (PTSD),  major  depression,  anxiety  disorders,  and  other 
post-deployment  adjustment  difficulties  affect  a  significant  number  of  veterans  returning 
from  Operation  Iraq  Freedom  and  Operation  Enduring  Freedom.  To  contend  with  this 
new  influx  of  veterans  suffering  from  the  psychological  aftermath  of  military  combat,  the 
VA  has  been  proactive,  including  commissioning  this  study  of  their  PTSD  health-care 
delivery  system,  with  the  objective  of  providing  the  best  care  in  the  most  efficient  manner 
possible  to  as  many  affected  veterans  as  possible. 

This  study  examines  PTSD  health-care  delivery  from  a  systems  engineering 
perspective.  First,  a  comprehensive  study  of  the  system  of  PTSD  health-care  delivery 
was  completed.  This  included  identification  and  analysis  of  the  mission  and  the 
environments,  identification  and  decomposition  of  functional  requirements,  the 
quantification  of  system  processes,  and  the  identification  of  systems  attributes  and 
measures. 

The  main  functions  of  the  system  were  determined  to  be:  “cure  people”  and 
“continuously  improve.”  Under  cure  people ,  the  main  work  of  PTSD  occurs.  The  sub¬ 
functions  for  cure  people  are  receive  patient,  assess  patient,  prepare  patient  for  treatment, 
provide  treatment,  and  disengage.  Under  continuously  improve  are  the  administrative 
sub-functions  maintain  patient  history,  monitor  performance,  and  adjust  process. 

The  characterization  of  the  system  was  completed  with  the  mapping  of  functions 
to  form  and  the  creation  of  a  cross-functional  flowchart. 

Armed  with  this  information,  the  work  of  creating  a  usable  model  of  the  system 
began  to  answer  the  research  question,  “For  the  system  of  PTSD  Health-care  delivery, 
what  are  optimal  factors  that  will  maximize  efficiency,  capacity,  and  quality?”  State-of- 
the-art  tools,  including  ExtendSim  modeling/simulation  software  and  JMP  analysis 
software  were  employed  (JMP  Ver  9.0.0  and  ExtendSim  Suite  Ver.  8.0.1). 

Each  subfunction  was  decomposed  into  the  major  tasks  required  to  support  the 

functional  requirement.  Then,  over  numerous  discussions  with  the  sponsor,  the 

xvii 


parameters  of  each  task  were  determined  in  sufficient  detail  so  that  they  could  be 
represented  in  the  ExtendSim  model.  ExtendSim  8  was  used,  which  provides  a  powerful 
tool  (Scenario  Manager)  for  the  design  of  experiments. 

Using  Scenario  Manager,  an  experiment  was  designed  around  eight  factors  and 
four  responses.  Six  factors  were  evaluated  over  three  levels,  and  the  remaining  two 
factors  were  evaluated  over  two  levels. 

The  eight  factors  of  the  system  are  number  of  health  care  providers,  provider 
intensity  (number  of  providers  required  per  patient  encounter),  group  sizes  for  two 
separate  forms  of  therapy,  number  of  sessions  provided  to  prepare  patients  for  intense 
therapy,  patient  cancellation  rate,  encounter  failure  rate,  and  patient  drop  rate. 

The  four  responses  were  provider  utilization  (measures  efficiency),  patient 
throughput  (measures  capacity),  average  treatment  duration  (measures  timeliness)  and 
average  time-between-encounters  (also  measures  timeliness).  For  this  thesis,  timeliness 
represents  quality  of  service  in  the  delivery  of  PTSD  health  care.  Maximizing  quality 
equates  to  providing  the  best  care.  From  the  standpoint  of  measuring  quality,  important 
future  work  would  incorporate  efficacy  for  the  various  treatment  paths  into  the  model. 

The  resulting  ExtendSim  model  was  programmed  to  simulate  260  weeks 
(approximately  five  years)  of  system  operation.  The  amount  of  time  allowed  for  the 
system  to  reach  equilibrium  averaged  26  weeks  (approximately  10%  of  the  simulation 
time).  Note  that  a  full-factorial  experiment  design  would  include  (36)(22)  =  2916 
scenarios.  With  three  trials  per  scenario,  there  are  8748  separate  trials.  This  would  not 
have  been  practicable  without  the  Scenario  Manager  feature  of  ExtendSim  8. 

The  results  were  automatically  recorded  to  a  table  within  Scenario  Manager. 
Scenario  Manager  then  exported  the  table  to  JMP  for  analysis. 

Using  the  Fit  Model  platform  within  JMP  and  invoking  the  Prediction  Profiler,  a 
response  curve  was  generated  for  each  factor-response  combination.  Then,  a  desirability 
function  was  created  for  each  response.  Executing  the  maximize  desirability  command 


within  JMP  produced  a  set  of  eight  optimized  system  factors,  which  maximize  the 
desirability  of  four  system  performance  measures  that  define  the  efficiency,  capacity,  and 
timeliness  of  the  system. 

The  improvements  in  system  performance  gained  by  using  optimal  factors  over 
the  base  case  were  notable.  These  improvements  are  summarized  in  Table  1.  Note  that 
all  system  attributes  are  improved,  and  that  an  80%  improvement  in  overall  system 
desirability  can  be  theoretically  achieved. 


Attribute 

Measured  Response 

Base 

Optim 

Delta 

%  Improve 

Efficient 

Provider  Utilization 

0.61 

0.88 

0.27 

44% 

Capacity 

Throughput  (patients) 

270 

310 

40 

15% 

Timely 

Treatment  Duration  (weeks) 

17.6 

10.0 

-7.6 

43% 

Timely 

Time  Between  Encounters  (weeks) 

2.30 

1.58 

-0.72 

31% 

Overall  System  Desirability 

0.49 

0.88 

0.39 

80% 

Table  1.  System  improvements  gained  using  optimized  factors  over  base  case 


These  improvements  are  obtainable  by  altering  the  system  factors  from  base  to 
optimal  as  described  in  Table  2.  Note  that  optimal  factors  for  provider  count,  scaffolding 
session  count,  cancellation  rate,  failure  rate,  and  drop  rate  were  expected,  while  those 
for  provider  intensity  and  CPT/CM  group  size  were  not  expected. 
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# 

Factor 

Base 

Optim 

1 

Provider  Count 

2 

2 

2 

Provider  Intensity 

Base 

Base 

3 

CPT  Group  Size 

9 

1 

4 

CM  Group  Size 

9 

1 

5 

Scaffolding  Session  Count 

4 

3 

6 

Cancellation  Rate 

Base 

Less 

7 

Failure  Rate 

Base 

Least 

8 

Drop  Rate 

Base 

Less 

Table  2.  Comparison  of  base  and  optimal  factors 

We  argue  that  these  models  can  and  should  be  used  as  a  platform  for  future  work 
in  this  area  of  study.  They  are  valuable  tools  for  real-world  decision-making.  Enhancing 
the  accuracy  and  scope  of  these  models  will  speed  the  rate  of  improvement  to  the  overall 
system  of  PTSD  health-care  delivery. 
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INTRODUCTION 


What  cannot  be  talked  about  cannot  be  put  to  rest.  And  if  it  is  not,  the 
wounds  will  fester  from  generation  to  generation. 

— Bruno  Bettelheim 


A.  BACKGROUND 

Post-Traumatic  Stress  Disorder  (PTSD),  major  depression,  anxiety  disorders,  and 
other  post-deployment  adjustment  difficulties  affect  a  significant  number  of  veterans 
returning  from  Operation  Iraq  Freedom  (OIF)  and  Operation  Enduring  Freedom  (OEF). 
For  example,  it  is  estimated  that  up  to  28%  of  those  who  served  in  OIF/OEF  would  meet 
the  diagnostic  criteria  of  PTSD  if  broader  screening  criteria  were  used  (Hoge,  Castro, 
Messer,  McGurk,  Cotting,  &  Koffman,  2004). 

The  VA  has  taken  a  proactive  approach  to  dealing  with  a  new  influx  of  veterans 
suffering  from  the  psychological  aftermaths  of  military  combat.  These  measures  have 
included  expanding  outreach  efforts  to  veterans  in  the  community,  integrating  mental 
health  care  within  the  primary  care  setting  to  decrease  stigma  and  improve  access,  and 
instituting  universal  mental  health  screenings.  According  to  statistics  gathered  from  the 
Veterans  Health  Information  Systems  and  Technology  Architecture  (VistA),  within  the 
Palo  Alto  VA  Health-care  system  (HCS),  approximately  80  veterans  screen  positive  for 
PTSD  per  month  of  which  72%  have  not  had  mental  health  care  within  the  last  two  years. 
These  numbers  indicate  that  accurate,  comprehensive  tracking  of  patients  from  diagnosis 
to  outpatient  care  is  integral  part  of  helping  these  veterans. 

Treatment  options  in  VA  for  patients  with  PTSD  range  widely  in  duration  and 
clinician  involvement.  They  include:  include  psychoeducation,  motivational 
interviewing/behavioral  activation,  web-based  self-help,  time-limited  group  therapy, 
short-term  individual  therapy  and  psychopharmacological  treatments  (DCOE  for 
Psychological  Health  &  Traumatic  Brain  Injury,  2011). 
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In  terms  of  health-care  delivery,  the  goal  is  to  provide  the  most  appropriate, 
effective  and  least  intrusive  intervention  as  soon  as  possible  to  the  greatest  number  of 
affected  individuals  possible.  For  patients  with  moderate  to  severe  PTSD  whom  do  not 
respond  to  very  brief  interventions,  forms  of  individual  therapy  known  as  Prolonged- 
Exposure  (PE)  and  Cognitive  Processing  Therapy  (CPT)  are  currently  the  “gold 
standards.”  Both  of  these  therapies  have  significant  amounts  of  empirical  support  for 
their  effectiveness  (Monson,  Schnurr,  Resick,  Friedman,  Young-Xu,  &  Stevens  2006) 
(Schnurr,  Friedman,  Engel,  Foa,  Shea,  Chow,  Resick,  Thurston,  Orsillo,  Haug,  Turner,  & 
Bernardy  2007).  Both  therapies,  however,  are  time-intensive  (e.g.,  PE  is  most  often 
administered  in  twelve,  90  minute  sessions).  Like  all  therapies,  there  is  a  variable 
response  to  exposure-based  therapies  and  many  patients  will  require  continuing  mental 
health  care  even  after  the  best  available  interventions  are  administered.  And,  acuity  of 
presentation  will,  of  course,  factor  into  clinical  decision-making  regarding 
frequency/intensity  of  provider  intervention. 

To  respond  to  new  guidelines  mandating  the  provision  of  state-of-the-art, 
research-based  PTSD  intervention,  mental  health  staffing  has  been  expanded,  but 
increased  staffing  alone  will  be  insufficient  without  other,  more  systemic  changes  in 
clinical  practice  (Department  of  Veterans  Affairs,  2010). 

These  changes  include  improvements  in  office  efficiency  to  enhance  timely 
access  of  care,  track  patient  status  from  inpatient  to  outpatient  care,  improved 
coordination  of  care  between  providers,  decreases  in  treatment  barriers,  increases  in 
effective  initial  treatment  assignments,  and  enhanced  efficiency  of  chronic  care  to  reduce 
staff  burden  and  maximizing  staff  productivity.  The  objective  is  to  provide  the  best  care 
in  the  most  ejficient  manner  possible  to  as  many  ajfected  veterans  as  possible.  However, 
without  a  systems  approach  to  this  goal,  it  is  unlikely  to  be  achieved. 

Further,  challenges  inherent  to  the  treatment  of  PTSD  in  veterans  (e.g.,  resistance 
to  change  due  to  identity  issues  or  to  compensation  seeking)  must  be  factored  into  any 
algorithm  addressing  system  performance  in  this  arena. 
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B.  RESEARCH  QUESTIONS 


1.  Main  Research  Question 

While  the  answer  may  be  somewhat  elusive,  the  question  to  be  asked  is  readily 
apparent.  In  short,  it  is,  “For  the  system  of  PTSD  Health-care  delivery,  what  are  optimal 
factors  that  will  maximize  efficiency,  capacity,  and  quality?” 

2.  Supporting  Research  Questions 

To  answer  the  main  research  question,  it  will  be  necessary  to  define  some  terms. 
Specifically,  the  definitions  of  efficiency ,  capacity,  and  quality  must  be  defined. 
Therefore  the  supporting  research  questions  are: 

•  What  defines  efficiency  in  PTSD  Health-care  delivery? 

•  What  defines  capacity  in  PTSD  Health-care  delivery? 

•  What  defines  quality  in  PTSD  Health-care  delivery? 

C.  SYSTEMS  ENGINEERING  APPROACH 

A  systems  engineering  (SE)  methodology  provides  structure  through  which 
complex  systems  may  be  examined.  In  a  purely  developmental  project,  the  starting  point 
would  begin  with  a  definition  of  the  problem  to  be  solved  or  a  requirement  to  be  met.  In 
a  re-engineering  project,  this  beginning  point  is  a  requirement  to  improve  the 
performance  of  a  complex  system  already  in  place.  In  this  case,  “system”  refers  to 
people,  technology,  policy,  tools  and  techniques,  which  are  interrelated  for  the  purpose  of 
doing  the  work  required  to  meet  required  measures  of  performance. 

The  development  of  a  working  model,  or  simulation,  reveals  the  specific  factors 
to  which  the  system  is  most  sensitive.  The  assumption  is  that  PTSD  health  care  includes 
many  aspects  of  a  dynamic  system  that  are  best  understood  in  a  working  simulation  that 
includes  the  relationships  and  interactions  between  all  of  the  working  components  of  the 
system. 
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While  the  SE  approach  is  broadly  methodical  and  leads  to  a  reliable  solution, 
there  remains  an  infinite  variety  within  the  path  of  the  research,  due  to  the  myriad  of  tools 
available  to  today’s  systems  engineer.  As  part  of  the  research  work,  an  investigation  of 
these  tools  and  their  suitability  to  the  problem  was  accomplished.  In  this  section,  a 
chronology  of  the  research  is  provided.  Some  of  this  narrative  describes  that  of  the 
overarching  SE  approach,  and  some  of  it  is  the  investigation  of  various  tools  and  their 
suitability  to  this  particular  project. 

1.  Define  the  Problem 

The  problem  presented  itself  during  several  discussions  with  principal 
investigators  from  the  VA:  Drs.  Josef  Ruzek  and  Steven  Lindley.  The  discussions  were 
summarized  in  the  paragraph  entitled  “Background”  and  the  problem  was  consolidated 
into  the  research  questions  presented  previously. 

2.  Identify  and  Analyze  Mission  and  Environments 

The  next  phase  of  the  systems  engineering  approach  was  applied  to  the 
investigation  of  the  work  of  the  PTSD  health  care  providers  within  the  environment  of 
the  VA  Health-care  system  (VAHCS).  As  the  VAHCS  is  distributed  nation-wide  across 
dozens  of  clinics,  a  single  clinic  was  chosen  to  represent  the  system. 

The  Monterey  Community-Based  Outpatient  Clinic  (CBOC)  was  an  obvious 
choice  as  the  subject  for  case  study.  The  clinic  serves  the  populations  of  the  Monterey 
peninsula,  south  to  Big  Sur,  CA,  and  north  to  Gilroy,  CA.  The  clinic  served  1480  mental 
health  and  5597  primary  care  patients  in  FY  2008.  In  terms  of  patients  served  and 
staffing  levels,  its  falls  into  a  “medium”  size  range  for  clinics  in  VAPAHCS.  Similar  to 
most  of  the  VA  clinics,  it  is  geographically  isolated  from  residential  and  other  specialty 
treatment  providers.  It  is  the  site  of  a  PTSD  Clinical  Team  (PCT),  comprised  of  a 
psychologist,  a  social  worker,  and  a  recreational  therapist.  These  factors  make  it 
representative  of  other  VAPAHCS  CBOCs,  and  appropriate  to  serve  as  the  test  site  for 
the  project.  In  addition,  it  treats  a  diverse  population:  former  military,  active  duty, 
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National  Guard  and  Reserve  personnel.  It  is  approximately  5  miles  from  the  Naval 
Postgraduate  School  (NPS),  making  it  a  logical  choice  due  to  its  proximity. 

Data  was  gathered  for  the  purpose  of  defining  specific  problems  apart  from 
systems -related  symptoms.  Types  of  data  collected  included  those  obtained  through 
interview  and  observation,  acquired  from  people  who  are  engaged  in  work  within  the 
system.  A  dozen  health  care  providers  were  interviewed.  Each  provider  was  asked  to 
relate  their  typical  workday,  the  segment  of  the  PTSD  population  they  served  and  in  what 
capacity  they  served  them,  and  specific  challenges  that  are  associated  with  their  work. 

3.  Identify  and  Decompose  Functional  Requirements 

Categorizing  functional  areas  and  their  relationships  to  people,  technology, 
policy,  or  health  care  schemas  was  necessary  for  development  of  the  working  model  of 
the  system.  The  process  of  codifying  the  basic  functions  of  the  system  began  as  a 
distillation  of  the  interview  notes.  Functions  were  then  decomposed  and  presented  to  the 
VA  for  validation  prior  to  the  commencement  of  model  development  efforts.  Figure  1 
shows  the  results  of  the  functional  decomposition. 


Figure  1 .  Functional  decomposition  of  PTSD  health  care 
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The  functional  requirements  of  a  system  that  delivers  PTSD  health  care  can  be 
divided  into  two  broad  categories.  The  first  is  the  function  of  curing  people  of  their 
PTSD.  The  word  “cure”  may  be  a  misnomer,  because  most  people  will  never  become 
fully  cured;  this  is  due  to  the  nature  of  the  disorder.  “Treat”  could  be  substituted  for 
“cure,”  but  the  ultimate  desire  is  for  people  to  lead  a  PTSD-free  life,  so  “cure”  is  used. 

The  second  major  functional  category  is  to  continuously  improve.  This  category 
contains  all  of  the  administrative  functions  that  are  not  directly  related  to  providing 
treatment  and  curing  people.  The  title  of  this  functional  category  was  specifically  chosen 
to  represent  the  true  purpose  of  the  sub-functions  that  it  contains.  Of  course,  PTSD 
therapy  can  be  delivered  without  the  tedium  of  tracking  patients  (either  individually  or  on 
the  aggregate)  but,  to  gain  improvements  to  the  system  these  things  must  be  done. 
Moreover,  collecting  data  but  not  using  it  in  a  meaningful  way  to  improve  the  success 
rate  of  PTSD  health  care  (i.e.,  refining  the  correlation  between  key  factors  and  measures) 
would  not  be  responsible  to  the  overarching  function  of  delivering  PTSD  health  care. 
Similarly,  collecting  the  wrong  or  insufficient  information  does  not  help  to  advance  the 
system  performance  and  therefore  does  not  support  the  overarching  function.  Lastly,  the 
information,  even  if  correct  and  complete,  must  be  easily  accessible  and  readily  usable. 

4.  Quantify  Processes 

Considerable  time  was  spent  in  consultation  with  Mr.  Harley  Barber,  a  VA 
information  system  specialist,  to  gather  baseline  statistics  of  the  system.  In  the  process,  it 
was  discovered  that  the  VistA  information  system  is  not  (currently)  adequately  tracking 
PTSD  system  metrics. 

Data  was  captured  (with  some  level  of  difficulty)  that  characterized  the  system 
parameters.  Aggregate  statistics  were  used  for  model  development.  As  there  is 
considerable  variety,  both  in  the  severity  of  PTSD  from  patient  to  patient  and  the 
treatment  paths  prescribed,  consultation  with  the  research  sponsor  was  necessary  to  form 
a  backdrop  to  the  statistics.  The  problem  was  compounded  by  the  efforts  to  transform 
and  standardize  treatment  paths.  In  other  words,  should  a  model  be  built  to  represent  the 
current  state  of  a  system  in  transition  (a  very  complicated  process),  or  should  it  be  based 
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on  the  desired  end-state  of  the  system  (a  more  achievable,  and  possibly  more  useful 
goal)?  The  decision  was  made  in  favor  of  the  latter. 

5.  Define  System  Attributes  and  Measures 

Recall  the  main  research  question:  For  the  system  of  PTSD  Health-care  delivery, 
what  are  optimal  factors  that  will  maximize  efficiency,  capacity,  and  quality?  The  key 
responses  are  in  the  realms  of  efficiency,  capacity,  and  quality. 

Efficiency  implies  efficiency  of  resources  used  in  the  system.  The  resources  in 
PTSD  health  care  are  mainly  human  resources  (viz.,  health  care  providers).  Efficiency  in 
the  use  of  human  resources  can  be  measured  in  Utilization.  Simply  put,  utilization  is  the 
percentage  of  the  providers’  available  time  that  is  spent  engaged  in  the  provision  of 
PTSD  health  care. 

Capacity  is  easily  measured  as  the  number  of  patients  processed  by  the  system 
over  a  period  of  time. 

Standards  of  quality  are  measured  from  the  patient’s  perspective.  A  patient 
suffering  from  PTSD  might  judge  quality  by  the  frequency  of  provider  contact  and  the 
speed  of  recovery.  For  instance,  if  a  treatment  plan  calls  for  provider-patient  contact  on  a 
weekly  basis,  but  because  the  clinic  is  understaffed,  the  patient  sees  the  provider  every 
three,  the  quality  of  care  might  be  deemed  to  be  low.  In  this  case,  the  situation  could  lead 
to  patient  disillusionment  or  apathy,  resulting  in  increased  patient  drop  rates,  for  example. 
The  same  argument  applies  to  speed  of  recovery. 

Provider  contact  frequency  is  easily  measured  in  the  model  by  tracking  and 
averaging  the  times-between-encounters.  Speed  of  recovery  can  be  measured  in  the 
model  by  tracking  and  averaging  the  delta  of  entry  to  and  exit  from  the  system  for  each 
patient.  These  are  both  measurements  that  quantify  an  attribute  of  the  system  that  could 
be  called  timeliness.  Therefore,  quality ,  in  one  sense  of  the  case  of  PTSD  health  care,  is 
defined  by  how  timely  the  care  is. 

Timeliness  is  not  the  only  quality  attribute  of  PTSD  health-care  delivery.  If  it 
were,  that  would  assume  that  every  patient  exiting  the  system  has  the  same  level  of 
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recovery  from  his  or  her  disorder.  Future  work  should  compare  the  entrance  and  exit 
PTSD  scores  from  standardized  psychological  tests  and  incorporate  the  efficacy  of 
treatment  plans  into  the  model. 

6.  Construct  Model 

A  key  goal  of  the  research  was  to  construct  a  working  model  of  the  system.  This 
task  was  attempted  using  a  variety  of  tools.  VDT  Power  and  Excel  both  produced 
working  models,  but  ultimately  ExtendSim  provided  the  most  robust  model  with  the 
greatest  flexibility. 


a.  VDT  Power 

A  model  was  built  using  VDT  Power,  given  workload  as  stated  by  the 
providers.  VDT  Power  is  an  excellent  tool  for  single,  start-to-finish  projects,  particularly 
when  inter-functional  communication  is  important  to  the  performance  of  the  system.  It 
was  difficult  to  apply  VDT  to  multiple,  asynchronous,  and  disparate  projects  (viz., 
individual  patients  receiving  individualized  treatment  in  overlapping  time  domains). 
VDT  Power  focuses  on  “information  flow  physics”  in  projects  where  “all  activities  in  the 
project  can  be  predefined”  (Levitt,  2009). 

b.  Microsoft  Excel 

A  standard  tool  for  smaller  problems  in  operations  research  is  Microsoft 
Excel,  in  conjunction  with  the  Solver  add-in  by  Frontline  Systems,  Inc.  An  Excel  model 
of  the  system  was  constructed  that  incorporated  many  of  the  system  parameters.  Using 
Solver,  the  model  was  optimized  for  lowest  provider  cost.  When  multiple  responses  are 
presented,  as  in  the  PTSD  heath  care  system,  use  of  Solver  and  Excel  becomes  tedious 
and  impracticable.  For  this  reason,  the  Excel  model  was  abandoned  and  construction  in 
ExtendSim  began. 
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c.  ExtendSim 

For  this  modeling  effort,  ExtendSim  presented  itself  as  the  most 
appropriate  tool.  ExtendSim  is  a  software  product  that  enables  the  building-block  style 
creation  of  discrete  event  models  that  represent  real-life  processes.  ExtendSim  supports 
the  creation  of  shift  schedules,  which  are  useful  in  describing  the  availability  of  health 
care  providers.  The  software  supports  database  communication,  allowing  the  user  to  pre¬ 
define  a  set  of  experimental  parameters  from  one  set  of  tables,  and  then  record  the  results 
to  another  set  of  tables.  In  addition,  version  8  of  ExtendSim  presents  excellent 
interoperability  with  the  JMP  analysis  tool,  which  became  an  important  part  of  this 
research. 

The  modeling  process  began  as  a  simple  approximation  of  the  functional 
requirements.  During  numerous  conferences  with  the  research  physicians  from  the  VA, 
and  through  analysis  of  encounter  data  provided  by  the  VA,  the  model  was  quantified  and 
refined  until,  after  several  iterations,  it  was  accepted  as  a  useful  approximation  of  the 
actual  system.  The  resulting  model  is  a  dynamic  simulation  that  mathematically 
describes  the  behavior  of  the  system  of  PTSD  health-care  delivery  over  time. 

A  version  of  the  model  was  constructed  in  version  7  of  ExtendSim.  The 
model  was  then  upgraded  to  version  8,  which  greatly  improved  the  design  of  experiments 
functionality. 

7.  Verify  Model 

The  ExtendSim  model  was  verified  by  comparing  the  results  of  one  scenario  with 
those  obtained  through  manual  (Excel)  number-crunching.  The  model  was  deemed  to  be 
mathematically  accurate. 
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8. 


Design  of  Experiments  (DOE)  and  Analysis  of  Results 


a.  Determine  Design  Factors  and  Levels 

Design  factors  became  evident  in  the  model  as  the  parameters  of  the 
system  functions  that  might  be  considered  “controllable”  in  the  real  world.  For  each 
factor,  a  set  of  levels  was  designed,  under  consultation  with  VA  physicians.  These  levels 
represented  incremental,  obtainable,  improvements  to  the  real-world  system. 

b.  Select  Orthogonal  Array 

The  number  of  combinations  of  factors  and  levels  being  too  high  for  an 
exhaustive,  full-factorial,  manual  execution,  a  suitable  orthogonal  array  was  found  from 
listings  of  predesigned  arrays.  With  an  orthogonal  array,  a  subset  of  level  combinations 
was  required  that  would  avoid  significant  loss  of  experiment  integrity.  A  predesigned 
orthogonal  array  was  chosen  that  would  provide  24  factor  scenarios. 

c.  Probabilistic  Design 

Included  in  the  DOE  are  probabilistic  design  factors.  In  the  simplest 
terms,  probabilistic  design  allows  the  factors  to  become  more  than  numeric  variables. 
With  the  incorporation  of  probabilistic  design  into  the  model,  some  of  the  factors  are 
represented  by  distributions.  In  the  case  of  this  research  in  its  current  state,  all 
distributions  are  empirical.  Future  work  should  include  a  more  detailed  analysis  of  the 
data.  This  may  uncover  smoother  distributions  that  may  result  in  an  improved 
representation  of  reality. 
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d.  Conduct  Experiment  and  Analyze  Data  (ExtendSim  v7  and 
Excel ) 

Using  ExtendSim  version  7,  the  factor  levels  were  manually  manipulated. 
Multiple  (5)  trials  of  each  experimental  unit  were  completed,  to  ensure  that  the  results 
were  corrected  for  randomness.  After  each  trial,  the  resulting  values  for  the  measures 
were  manually  recorded  to  a  response  table.  The  manual  execution  of  the  experiment 
required  approximately  3.5  hours  to  complete. 

Then,  using  Excel  to  analyze  the  response  table,  for  each  experimental 
unit  and  for  each  measure,  an  average  of  the  5  trials  was  computed.  An  average  was 
computed  across  experimental  units,  within  each  parameter  and  level.  The  results, 
displayed  in  response  curves,  were  the  combined  main  effects  of  changing  parameter 
levels  on  the  system  performance  measures. 

e.  Program  and  Conduct  Experiment  ( ExtendSim  v8) 

The  tedious  process  of  conducting  the  experiment  was  automated  with  the 
implementation  of  the  model  in  version  8  of  ExtendSim,  which  increased  the 
experimental  budget  significantly.  Using  Scenario  Manager,  the  experiment  simply 
needed  to  be  programmed  into  ExtendSim.  Execution  was  automatic,  requiring  no 
human  interaction.  Analysis  of  the  resulting  table  of  responses  was  greatly  simplified 
using  Scenario  Manager’s  interoperability  with  JMP. 

As  result,  the  design  was  improved  to  a  full-factorial  experiment.  A  full- 
factorial  scheme  of  the  experiment’s  2916  design  points  was  executed,  at  three  trials 
each,  for  a  total  of  8748  runs  of  the  model.  Once  Scenario  Manager  was  fully 
programmed,  the  experiment  was  executed,  without  further  input,  over  a  period  of  four 
hours. 
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/.  Create  JMP  Model 


The  results  of  the  experiment  were  then  exported  to  JMP  for  analysis. 
After  some  adjustments  were  made  to  the  response  table  (viz.,  renaming  field  codes  and 
defining  sort  criteria  to  enhance  readability  of  the  results,)  the  JMP  model  was  created 
using  the  “Fit  Model”  platform. 

Next,  the  “Profiler”  platform  in  JMP  was  used  to  generate  a  matrix  of 
response  curves  for  each  response-factor  combination. 

The  last  step  in  the  analysis  was  to  set,  within  JMP,  the  desired  setting  for 
each  response.  JMP  created  a  set  of  desirability  functions  using  those  factors.  Upon 
giving  JMP  the  command  to  “maximize  desirability,”  a  list  of  optimized  factors  was 
generated.  The  results  were  interesting,  as  will  be  shown. 

D.  SCOPE  OF  THIS  THESIS 

The  scope  of  this  research  is  primarily  to  examine  a  generalized  system  of  PTSD 
health-care  delivery.  The  generalized  system  is  represented  by  the  ExtendSim  model, 
and  ultimately  in  the  JMP  model,  either  of  which  can  be  manipulated  to  discover  the 
outcomes  of  various  factors. 

It  is  important  to  emphasize  that  the  models  presented  in  this  thesis  are 
representations  of  reality  made  within  the  filter  of  a  broad  set  of  assumptions.  It  is 
acknowledged  that  these  assumptions  may  be  in  need  of  fine-tuning,  which  should  be  the 
subject  of  future  work.  In  the  process,  the  model  will  require  initial  validation  and 
subsequent  validation  at  each  iteration  of  major  improvement. 

E.  BENEFITS  OF  THIS  STUDY 

Using  the  products  of  this  research,  the  VA  decision  maker  can  discover  the 
factors  that  have  the  greatest  impact  on  PTSD  health-care  delivery,  which  will  allow  him 
to  make  improvements  to  the  system  in  the  most  efficient  manner,  given  the  limited 
resources  available.  The  end  result  is  a  more  rapidly-improved  system  that  provides 
better,  more  efficiently-delivered  health  care  serving  more  patients  with  PTSD. 
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In  addition,  this  systems  engineering-based  research  will  inform  the  VA  decision 
maker  of  the  data  that  is  required  to  be  collected  for  successful  monitoring  of  the  PTSD 
Health-care  system.  Knowing  this,  the  VistA  Information  system  can  be  replaced  or 
modified  to  collect  this  data  and  present  the  resulting  information  in  a  more  usable  and 
readily-available  form,  thereby  improving  the  function  of  system  monitoring  (and 
ultimately  increasing  the  speed  of  system  improvement). 

Future  work  can  be  applied  to  enhancing  the  models  to  bring  them  to  a  higher 
degree  of  alignment  with  reality.  This  will  cause  the  models  to  become  more  complex, 
but  it  will  increase  the  ability  of  the  decision  maker  to  more  finely  tune  the  real-life 
system. 
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II.  SYSTEM  DESCRIPTION 


A.  OVERVIEW 


In  the  simplest  of  terms,  the  system  of  providing  PTSD  health  care  involves 
untreated  patients  entering  the  system  and  cured  patients  exiting  the  system.  Along  the 
way,  health  care  providers  are  used  as  a  resource.  Figure  2  shows  a  very  basic  flowchart 
diagramming  this  system. 


Providers 


Figure  2.  Basic  flowchart  for  PTSD  health-care  delivery 


Expanding  upon  the  basic  flowchart,  the  functional  decomposition  of  curing 
people  (Figure  1)  is  incorporated,  resulting  in  Figure  3. 


Diagnoses  & 
Treatment  Plans 


Providers 

Figure  3.  Expanded  flowchart  for  PTSD  health-care  delivery 
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Note  that  providers  are  involved  in  all  of  the  major  sub-functions  of  curing 
people.  This  is  the  nature  of  PTSD  health  care.  To  further  emphasize  this  point, 
functions  were  mapped  to  physical  components  of  the  system  in  Table  3.  Note  that  all  of 
the  functions  require  a  human  health  care  provider  or  staff.  Second  to  human  health  care 
provider  in  system  intensity  is  the  medical  information  system. 


Receive 

Assess 

Prepare 

Treat 

Disengage 

Continuously 

Improve 

Receive  Referral 

Schedule  Assessment 

Perform  Assessment 

Establish  Diagnosis 

Create  Treatment  Plan 

Provide  Scaffolding 

Provide  PE 

Provide  CPT 

Provide  Care  Management 

Provide  Medication  Management 

Provide  Follow-on  Care 

Discharge  Case 

Maintain  Patient  History 

Measure  Performance  &  Effectiveness 

Adjust  Process 

Telephone  Reception 

X 

Reception  Desk 

X 

X 

Medical  Info  System 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Human  Provider/Staff 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

PTSD  Website 

X 

X 

X 

PTSD  Policy 

X 

Table  3.  Mapping  functions  to  physical  components 


As  described  in  the  introduction,  the  system  of  delivering  PTSD  healthcare 
involves  two  major  functions:  curing  people  and  continuously  improving.  Supporting 
functions  for  curing  people  are:  receiving,  assessing,  and  preparing  the  patient,  followed 
by  providing  treatment  and  disengaging.  Under  continuously  improve ,  the  functions  are 
maintain  patient  history,  monitor  performance  and  effectiveness,  and  adjust  process. 
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Figure  4  integrates  the  sub-functions  of  cure  people ,  as  they  are  shown  in  Figure  1 
into  a  business  process  flowchart.  The  result  is  a  cross-functional  flowchart  that  provides 
a  clear  chronology  of  the  major  activities  involved  with  delivering  PTSD  health  care 
(Microsoft  Visio  Premium,  2010). 


Figure  4.  Cross-functional  flowchart  for  the  system  of  PTSD  health-care  delivery 


B.  RECEIVE  PATIENT 

Receiving  a  patient  involves  receiving  a  referral  and  scheduling  an  assessment. 

1.  Receive  Referral 

Referrals  can  come  from  a  variety  of  sources.  As  PTSD  screening  has  been 
integrated  into  the  primary  care  setting,  a  majority  of  referrals  come  from  this  source. 
Other  referral  sources  include  self-referral,  referrals  from  psychiatric  care,  referrals  from 
law  enforcement,  etc. 
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2. 


Schedule  Assessment 


Scheduling  an  assessment  is  a  relatively  simple  administrative  function.  It 
involves  matching  a  new  patient  up  with  a  provider  for  an  in-depth  assessment. 

C.  ASSESS  PATIENT 

1.  Perform  Assessment 

Most  patients  are  assessed  over  a  single  90-minute  one-on-one  interview  session 
with  a  provider.  Some  patients  require  more  than  one  session  to  complete  the 
assessment. 

2.  Establish  Diagnosis 

Providers  establish  the  diagnosis  of  PTSD  during  and  after  the  90-minute 
assessment.  The  severity  of  the  diagnosis  is  also  established.  The  severity  will  influence 
the  treatment  plan  prescription.  If  the  patient  is  determined  not  to  have  PTSD,  no  further 
action  will  be  taken. 

3.  Create  Treatment  Plan 

The  treatment  plan  is  created  once  the  diagnosis  is  set.  This  involves 
documenting  the  diagnosis  in  the  medical  information  system.  It  may  involve  conferring 
with  other  qualified  providers  and/or  transferring  the  patient  to  a  different  provider. 

D.  PREPARE  PATIENT  FOR  TREATMENT 

1.  Scaffolding 

Some  patients  are  prescribed  PE  or  CPT  in  their  treatment  plan.  PE  and  CPT  are 
Evidence -Based  Therapies  (EBT).  EBT  are  intense  therapies,  designed  for  severe  cases 
of  PTSD.  These  therapies  require  a  significant  amount  of  commitment,  readiness,  and 
willingness  from  the  patient.  Accordingly,  not  all  patients  are  immediately  prepared  for 
EBT.  To  prepare  an  EBT  candidate  for  CPT  or  PE,  a  provider  will  meet  with  the  patient 
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for  an  indeterminate  number  of  encounters.  This  preparatory  phase  is  referred  to  in  this 
thesis  as  scaffolding.  The  term  is  not  widely  accepted,  but  generally  it  describes 
groundwork  that  is  required  before  EBT  can  begin.  It  consists  of  an  indeterminate 
number  of  sessions  with  a  provider,  focused  on  preparing  the  patient  for  the  rigorous, 
intense  therapy  that  is  EBT. 

If  a  patient  is  not  prescribed  EBT  in  their  treatment  plan,  they  generally  do  not 
require  scaffolding. 

E.  PROVIDE  TREATMENT 

All  therapies  are  intended  for  delivery  over  a  multiple  number  of  encounters,  with 
one  encounter  being  delivered  each  week  (except  medication  management,  which  is 
delivered  every  13  weeks).  The  actual  frequency  of  the  treatment  will  vary,  depending 
on  the  availability  of  a  provider,  the  availability  of  group  members,  and  whether  the 
patient  cancels  the  encounter  appointment.  Therefore,  the  frequency  indicated  for  the 
prescribed  treatment  is  to  be  viewed  as  a  maximum  frequency. 

Some  therapies  are  delivered  in  groups  (i.e.,  more  than  one  patient  per  provider,) 
and  some  treatment  plans  require  more  than  one  provider.  Group  size  and  provider 
intensity  and  their  incorporation  into  the  model  will  be  discussed  later. 

1.  Prolonged  Exposure 

PE  includes  psychoeducation,  breathing  retraining,  exposure  to  situations  that  the 
patient  has  related  to  the  trauma,  and  the  recalling/recounting  of  traumatic  events.  It  is  a 
12-session  program,  and  is  always  conducted  on  an  individual  basis  (group  size  =  1). 
Patient-provider  encounters  are  meant  to  be  90  minutes  each. 

2.  Cognitive  Processing  Therapy 

Cognitive  Processing  Therapy  is  designed  to  address  non-fear-related  emotions, 
the  context  of  the  meaning  elements  of  the  traumatic  memory,  and  examine  the  level  of 
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accommodation  that  has  been  made  for  the  traumatic  memory.  The  structure  of  CPT  is 
similar  to  that  of  PE,  except  that  CPT  may  be  provided  in  a  group  setting.  Group  size  for 
CPT  is  a  design  factor  for  the  system. 

3.  Care  Management 

Non-EBT  therapy  is  referred  to  as  care  management.  This  is  more  general 
psychotherapy,  focused  on  the  PTSD  symptoms  of  the  patient.  It  is  defined  by  both  the 
number  of  appointments  and  the  group  size.  Both  are  design  factors  for  the  system  and 
can  vary  widely  among  patients. 

4.  Medication  Management 

All  PTSD  patients  are  offered  medication  if  their  assessment  indicates  it.  Half  of 
these  patients  will  accept  the  medication.  Of  those  who  accept,  about  half  will  require 
traditional  (i.e.,  PE,  CPT,  or  care  management)  in  addition  to  medication  management. 

F.  DISENGAGE 

Most  PTSD  patients  will  live  with  some  level  of  their  disorder  for  the  rest  of  their 
lives.  For  this  model,  however,  all  patients  are  assumed  to  have  a  finite  treatment  plan,  as 
described  in  this  document.  Further  study  will  be  required  to  determine  more  realistic 
categories  for  the  methods  of  disengagement  from  (or  continuance  in)  the  PTSD  health¬ 
care  system. 

G.  CONTINUOUSLY  IMPROVE 

1.  Maintain  Patient  History 

The  medical  information  system  in  this  case  is  VistA.  VistA  is  a  legacy  system 
that  has  been  evolving  over  the  past  few  decades.  It  needs  considerable  improvement  or 
replacement  to  adequately  serve  the  function  of  delivering  PTSD  health  care. 
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2. 


Monitor  Performance  and  Effectiveness 


A  robust  medical  information  system  that  is  tailored  to  deliver  the  required 
measures  of  performance  and  effectiveness  is  a  necessary  component  to  sustain 
continuous  improvements  to  the  overall  system. 

3.  Adjust  Process 

Adjusting  the  process  involves  changing  PTSD  policy,  based  on  revised 
optimization  resulting  from  a  refined  model.  Improvements  made  in  this  manner  will 
improve  the  quality,  efficiency,  and  timeliness  of  the  PTSD  health-care  system. 
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III.  OPERATIONAL  MODEL 


A.  MODEL  DEVELOPMENT  AND  IMPLEMENTATION 

As  previously  noted,  the  main  simulation  platform  for  the  model  was  originally 
ExtendSim,  Version  7,  but  was  upgraded  to  Version  8  to  capitalize  on  new  features  that 
were  important  to  the  research.  The  description  of  model  development  and 
implementation  below  concerns  the  use  of  Version  8.  Screen  shots  of  the  final  version  of 
the  ExtendSim  model  are  provided  in  Appendix  A:  ExtendSim  Screen  Shots.  Next,  we 
describe  the  parameters  of  the  PTSD  health-care  delivery  system  as  they  are  used  in  the 
model.  We  will  also  highlight  the  factors  and  responses. 

1.  Providers 

a.  Work  Schedule 

Each  provider  works  40  hours  per  week  (Monday  through  Friday),  and 
receives  five  weeks  of  vacation  per  year.  Within  each  workday,  a  provider  works  eight 
hours,  but  three  of  those  hours  are  consumed  with  meetings  and  chart  work  (patient 
tracking).  This  leaves  five  hours  per  workday  available  for  patient  contact. 

b.  Provider  Quantity 

To  determine  the  effect  that  the  number  of  available  providers  has  on  the 
system,  the  provider  quantity  is  made  to  be  a  design  factor  of  the  experiment.  The  levels 
for  this  factor  are  two,  three,  and  four  providers. 

c.  Utilization 

Provider  utilization  is  a  measure  of  efficiency  for  the  system.  ExtendSim 
provides  automatic  calculation  of  resource  utilization.  Two  variations  of  utilization  are 
considered  for  this  research.  One  considers  patient  cancellations  as  utilized  time,  the 
other  considers  patient  cancellations  as  unutilized  (lost)  time.  This  technique  will  help 
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the  user  of  the  model  to  determine  the  scheduling  requirements  needed  to  obtain  a  desired 
provider  utilization  given  an  expected  patient  cancellation  rate. 

2.  Patients 

a.  Rate  of  Arrival 

Patients  are  assumed  to  arrive  at  a  rate  that  is  determined  by  an 
exponential  distribution,  with  a  mean  of  one  patient  every  four  calendar  days.  While  this 
parameter  is  adjustable  within  the  ExtendSim  model,  it  is  not  a  design  factor  of  the 
experiment. 


b.  Rate  of  Drop 

Patients  will  discontinue  treatment  for  a  variety  of  reasons.  The 
overarching  desire  is  that  all  patients  adhere  to  their  prescribed  treatment  plans,  but  it  is 
expected  that,  regardless  of  the  effort  made  to  retain  patients,  a  certain  drop  rate  will 
exist.  As  the  objective  is  to  provide  the  best  care  possible  to  as  many  patients  as  possible, 
and  given  that  there  exists  a  finite  amount  of  resources  available  with  which  to 
accomplish  this,  it  would  be  good  to  know  how  patient  drop  rate  affects  the  system. 

To  determine  its  effect  on  the  system,  patient  drop  rate  is  made  to  be  a 
probabilistic  design  factor  for  the  experiment.  There  are  two  levels  for  this  design  factor: 
base  case  and  less  drops.  Each  level  is  an  empirical  distribution  representing  the 
probability  of  a  patient  dropping  after  a  traditional  therapy  session.  The  levels  are 
defined  in  Table  4. 


After  Each  Traditional 
Therapy  Session,  the 
Probability  a  Patient  Will: 

Drop 

Adhere 

Base  Case 

0.02 

0.98 

Less  Drops 

0.01 

0.99 

Table  4.  Definition  of  probabilistic  design  levels  for  patient  drop  rate 
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Future  work  would  investigate  the  characteristics  of  the  patient  drop  rate 
to  develop  a  more  precise  modeling  of  this  activity.  Discovering  categories  of  drop 
reasons  and  linking  these  categories  to  events  within  the  model  would  help  significantly 
to  discover  the  optimal  application  of  resources  toward  the  reduction  of  the  overall  drop 
rate. 


3.  Assessments 

Within  the  ExtendSim  model,  a  patient  is  assessed  as  soon  as  a  provider  resource 
is  available.  Note,  in  its  current  state,  this  model  only  considers  one  resource:  the  PTSD 
health  care  provider.  In  addition,  all  providers  are  assumed  to  be  equally  qualified  and 
equally  appropriate  to  every  task.  Therefore,  staff  time  required  for  handling  the  referral 
and  scheduling  the  assessment  are  not  considered  in  the  model. 

Assessments  are  approximately  90  minutes  each,  and  require  a  single  provider 
resource  to  complete.  The  model  introduces  randomness  to  the  assessment  session 
duration  by  incorporating  a  triangular  distribution  with  a  minimum  of  80,  a  maximum  of 
100,  and  a  mean  of  90  minutes. 

Future  work  may  include  developing  multiple  human  resource  categories  (e.g., 
staff,  multiple  levels  of  qualification  within  the  provider  ranks,  etc.).  While  ExtendSim 
version  8  provides  a  new  feature  named  Advanced  Resource  Management  (ARM)  that 
will  prove  to  be  a  valuable  tool  for  this  effort,  it  is  not  used  in  this  thesis. 

4.  Establishing  Diagnoses 

In  this  model,  all  referrals  are  assumed  to  have  PTSD  in  some  form  that  will 
require  treatment  using  one  or  more  of  the  four  treatment  plans:  medication  management, 
care  management,  PE,  or  CPT. 

Future  work  may  include  a  bogus  referral  rate,  if  that  is  appropriate  to  the  model. 
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5. 


Distribution  of  Treatment  Plan  Types 


a.  General 

The  current  distribution  of  treatment  plans  among  the  patient  population  is 
assumed  to  be  known.  In  broad  terms,  there  are  three  treatment  paths:  a  patient  can  be 
treated  with  traditional  therapy  (viz.,  care  management,  PE,  or  CPT);  a  patient  can  be 
treated  with  medication  management  alone;  or  a  patient  can  be  treated  using  both 
approaches.  For  the  model  in  its  current  state,  the  following  distribution  is  applied.  This 
distribution  is  adjustable,  but  it  is  not  a  design  factor  for  the  experiment: 

•  50%  will  require  traditional  therapy  only 

•  25%  will  require  medication  management  only 

•  25%  will  require  a  combination  of  the  two 

b.  Traditional  Therapy 

Within  traditional  therapy,  there  are  two  possible  treatment  paths:  EBT  or 
care  management.  As  previously  stated,  EBT  is  intensive  therapy  for  severe  cases  of 
PTSD,  while  care  management  is  reserved  for  milder  cases.  For  the  model  in  its  current 
state,  the  following  distribution  is  applied.  This  distribution  is  adjustable,  but  it  is  not  a 
design  factor  for  the  experiment: 

•  27%  will  require  EBT 

•  73%  will  require  care  management 

c.  EBT 


Within  EBT,  there  are  two  possible  treatment  paths:  PE  or  CPT.  For  the 
model  in  its  current  state,  the  following  distribution  is  applied.  This  distribution  is 
adjustable,  but  it  is  not  a  design  factor  for  the  experiment: 

•  50%  will  require  PE 


50%  will  require  CPT 
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6. 


Scaffolding 


When  prescribing  EBT,  the  model  currently  assumes  that  some  quantity  of 
scaffolding  sessions  will  be  required,  as  previously  described.  To  determine  the  effect 
that  quantity  of  scaffolding  sessions  has  on  the  system.  It  is  included  as  a  design  factor  of 
the  experiment.  In  setting  this  parameter  as  a  design  factor,  the  assumption  is  made  that 
there  is  a  certain  amount  of  control  over  the  average  number  of  scaffolding  sessions  that 
are  provided  to  the  patient  population  that  is  prescribed  EBT.  The  levels  for  this  design 
factor  are  three,  six,  or  nine  sessions. 

Scaffolding  sessions  are  approximately  60  minutes  each,  and  require  a  single 
provider  resource  to  complete.  The  model  introduces  randomness  to  the  scaffolding 
session  duration  by  incorporating  a  triangular  distribution  with  a  minimum  of  50,  a 
maximum  of  70,  and  a  mean  of  60  minutes. 

7.  Provider  Intensity 

Provider  intensity  is  defined  as  the  number  of  providers  required,  on  average,  for 
a  given  traditional  therapy  session.  Currently,  a  majority  of  traditional  therapy  is 
provided  by  more  than  one  provider.  To  determine  its  effect  on  the  system,  provider 
intensity  is  made  to  be  a  probabilistic  design  factor  for  the  experiment.  There  are  three 
levels  for  this  design  factor:  base  case;  less  intensity;  and  least  intensity.  Each  case  is  a 
different  distribution  of  the  number  of  providers  required  for  a  traditional  therapy  session. 
The  probabilistic  design  levels  for  provider  intensity  are  defined  in  Table  5. 


%  of  Traditional  Therapy 
Sessions  Requiring: 

1  Provider 

2  Providers 

Base  Case 

25 

75 

Less  Intensity 

50 

50 

Least  Intensity 

75 

25 

Table  5.  Definition  of  probabilistic  design  levels  for  provider  intensity 
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8. 


Structure  of  Therapies 


a.  PE 

PE  is  set  in  the  model  as  consisting  of  12  weekly  sessions  in  a  nongroup 
(single  patient)  setting.  Session  duration  and  variability  are  identical  to  those  for 
assessments. 


b.  CPT 

CPT  is  set  in  the  model  as  consisting  of  12  weekly  sessions  in  either  a 
group  or  single  patient  setting.  To  determine  the  effect  of  CPT  group  size  on  the  system, 
it  is  made  to  be  a  design  factor  of  the  experiment.  The  levels  for  this  factor  are  groups  of 
one,  five,  and  nine  patients  per  group.  Session  duration  and  variability  for  CPT  sessions 
are  identical  to  those  for  assessments. 

c.  Care  Management 

Care  management  is  the  most  common  of  the  traditional  therapies  in  use. 
It  consists  of  4  weekly  sessions  in  either  a  group  or  single  patient  setting.  To  determine 
the  effect  of  care  management  group  size  on  the  system,  it  is  made  to  be  a  design  factor 
of  the  experiment.  The  levels  for  this  factor  are  groups  of  one,  five,  and  nine  patients  per 
group.  Session  duration  and  variability  for  care  management  sessions  are  identical  to 
those  for  assessments. 

d.  Medication  Management 

Medication  management  consists  of  quarterly  visits  with  a  provider  in  a 
single  patient/single  provider  setting,  with  each  session  lasting  approximately  30  minutes. 
The  model  introduces  randomness  to  the  medication  management  session  duration  by 
incorporating  a  triangular  distribution  with  a  minimum  of  20,  a  maximum  of  40,  and  a 
mean  of  30  minutes. 
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If  a  patient  is  undergoing  traditional  therapy  concurrent  with  medication 
management,  it  is  assumed  that  the  quarterly  medication  appointments  will  occur  within  a 
traditional  therapy  session  for  as  long  as  the  traditional  therapy  plan  continues.  When  the 
traditional  therapy  is  concluded,  separate  medication  management  appointments  are 
scheduled. 

In  reality,  patients  continue  medication  management  for  varying  lengths  of 
time.  For  the  model  in  its  current  state,  all  patients  are  assumed  to  continue  medication 
management  for  five  years.  At  the  end  of  five  years,  the  patient  is  disengaged. 

Future  work  would  investigate  the  characteristics  of  the  medication 
management  program  to  develop  a  more  precise  modeling  of  this  activity.  This  would 
require  modifications  to  the  VistA  medical  information  system,  as  it  is  not  currently  set 
up  to  track  this  information. 

9.  Patient-Provider  Encounters 

Patient-provider  encounters  are  subject  to  variability  completion  rate  due  to 
variability  in  rates  of  patient  cancellation  and  encounter  failure. 

a.  Patient  Cancellation 

Patients  will  cancel  their  scheduled  appointment  at  a  known  rate.  There 
are  two  basic  types  of  patient  cancellations:  those  that  are  made  sufficiently  in  advance  of 
the  scheduled  appointment  such  that  the  provider  may  be  rescheduled;  and  those  that  are 
not.  The  former  are  not  considered  in  this  model,  because  they  do  not  affect  provider 
utilization.  The  latter  are  important  to  the  system. 

To  determine  its  effect  on  the  system,  patient  cancellation  rate  is  made  to 
be  a  probabilistic  design  factor  for  the  experiment.  There  are  two  levels  for  this  design 
factor:  base  case  and  less  cancellation.  Each  level  is  an  empirical  distribution 
representing  the  probability  of  a  patient  cancelling  prior  to  a  traditional  therapy  session 
(with  insufficient  time  to  reschedule  the  provider).  The  probabilistic  design  levels  for 
patient  cancellation  rate  are  defined  in  Table  6. 
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Prior  to  Each  Traditional 
Therapy  Session,  the 
Probability  a  Patient  Will: 

Cancel 

Attend 

Base  Case 

0.18 

0.82 

Less  Cancellation 

0.09 

0.91 

Table  6.  Definition  of  probabilistic  design  levels  for  patient  cancellation  rate 

Future  work  would  investigate  the  characteristics  of  the  patient 
cancellation  rate  to  develop  a  more  precise  modeling  of  this  activity.  Discovering 
categories  of  cancellation  reasons  and  linking  these  categories  to  events  within  the  model 
would  help  significantly  to  discover  the  optimal  application  of  resources  toward  the 
reduction  of  the  overall  cancellation  rate. 

b.  Encounter  Failure 

Not  all  patient-provider  encounters  will  be  successful.  Assuming  there  is 
an  established  structure  to  traditional  therapy  that  must  be  completed  in  a  sequential 
manner  (i.e.,  completing  the  current  phase  is  a  prerequisite  to  proceeding  to  the  next 
phase,  etc.),  then  failing  to  meet  the  requirements  of  the  scheduled  encounter  will  result 
in  the  need  to  repeat  that  session. 

To  determine  its  effect  on  the  system,  encounter  failure  rate  is  made  to  be 
a  probabilistic  design  factor  for  the  experiment.  There  are  three  levels  for  this  design 
factor:  base  case ;  less  failure-,  and  least  failure.  Each  level  is  an  empirical  distribution 
representing  the  probability  of  the  failure  of  a  traditional  therapy  session.  The 
probabilistic  design  levels  for  encounter  failure  rate  are  defined  in  Table  7. 
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Probability  that  a 
Traditional  Therapy 
Session  Will: 

Fail 

Succeed 

Base  Case 

0.30 

0.70 

Less  Failure 

0.20 

0.80 

Least  Failure 

0.10 

0.90 

Table  7.  Definition  of  probabilistic  design  levels  for  encounter  failure  rate 

Future  work  would  investigate  the  characteristics  of  the  encounter  failure 
rate  to  develop  a  more  precise  modeling  of  this  phenomenon.  Discovering  categories  of 
reasons  for  failure  and  linking  these  categories  to  events  within  the  model  would  help 
significantly  to  discover  the  optimal  application  of  resources  toward  the  reduction  of  the 
overall  failure  rate. 

B.  SUMMARY  OF  FACTORS,  RESPONSES,  AND  EXPERIMENTAL 

DESIGN 

1.  Factors 

Experiment  factors  are  summarized  in  Table  8.  Note  that  a  full-factorial 
experiment  design  includes  (36)(22)  =  2916  scenarios.  With  three  trials  per  scenario, 
there  are  8748  separate  trials.  This  would  not  have  been  practicable  without  the  Scenario 
Manager  feature  of  Extends im  8. 
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# 

Factor 

Level  1 

Level  2 

Level  3 

1 

Provider  Count 

2 

3 

4 

2 

Provider  Intensity 

Base 

Less 

Least 

3 

CPT  Group  Size 

1 

5 

9 

4 

CM  Group  Size 

1 

5 

9 

5 

Scaffolding  Session  Count 

3 

6 

9 

6 

Cancellation  Rate 

Base 

Less 

-- 

7 

Failure  Rate 

Base 

Less 

Least 

8 

Drop  Rate 

Base 

Less 

-- 

Table  8.  Summary  of  experiment  factors 


2.  Responses 

Experiment  responses  are  summarized  in  Table  9. 


Attribute 

Measured  Response 

Definition 

Efficient 

Provider  Utilization 

%  of  available  provider  time 
engaged  in  PTSD  work 

Capacity 

Throughput 

Quantity  of  patients 
completed  over  time 

Timely 

Treatment  Duration 

Average  duration  of 
traditional  treatment  plan 

Timely 

Time  Between  Encounters 

Average  time  between 
traditional  therapy  encounters 

Table  9.  Summary  of  experiment  responses 


C.  RESULTS 

The  resulting  ExtendSim  model  was  programmed  to  simulate  260  weeks 
(approximately  five  years)  of  system  operation.  The  amount  of  time  allowed  for  the 
system  to  reach  equilibrium  averaged  26  weeks  (approximately  10%  of  the  simulation 
time).  The  experiment  executed  automatically  and  was  completed  in  approximately  four 
hours.  The  results  were  automatically  recorded  to  a  table  within  scenario  manager. 
Scenario  manager  then  exported  the  table  to  JMP  for  analysis. 
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IV.  ANALYSIS 


A.  JMP  MODEL 

Given  that  a  full-factorial  experiment  was  conducted  on  using  the  ExtendSim 
model,  JMP  was  able  to  generate  a  least  squares  fit  that  well  predicts  the  combined 
effects  of  the  model  factors.  For  parameter  estimates  and  prediction  plots  of  the  JMP 
model,  please  see  Appendix  B:  JMP  Model  Fit  Statistics. 

B.  PREDICTION  PROFILER 

The  prediction  profiler  (Figure  5)  displays  response  curves  for  each  factor.  A 
response  curve  is  the  predicted  response  as  one  variable  is  changed  while  the  others  are 
held  constant  at  the  current  values.  The  profiler  recalculates  the  predicted  responses  (in 
real  time)  as  the  value  of  a  factor  is  varied.  In  this  manner,  the  profiler  is  a  way  of  testing 
the  system  responses  due  to  changing  one  factor  at  a  time. 


33 


A  -  Prediction  Profiler 


Provider  CPT  Group  CM  Group  Scaff  Cancellation  Success  Lower  Provider 

Count  Size  Size  Session  Count  Rate  Rate  Drop  Rate  Intensity  Desirability 

Figure  5.  JMP  prediction  profiler  (including  maximized  desirability  functions) 


1.  Interpreting  Factors  and  Responses 

The  vertical  dotted  line  for  each  factor  shows  its  current  value  or  setting.  If  the 
factor  is  nominal,  the  X-axis  identifies  categories.  For  each  factor,  the  value  above  the 
name  is  its  current  value.  In  JMP,  the  current  value  can  be  changed  by  clicking  in  the 
graph  or  by  dragging  the  dotted  line  to  a  new  value.  As  shown,  all  the  factors  are  in  the 
optimal  factors  to  maximize  desirability,  which  will  be  discussed  later. 

The  horizontal  dotted  lines  show  the  current  predicted  values  for  each  response 
given  the  current  values  of  the  factors.  The  current  predicted  value  is  shown  to  the  right 
(underneath)  the  response  name.  The  95%  confidence  interval  on  the  mean  response  is 
shown  underneath  the  current  predicted  value. 
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2. 


Interpreting  System  Sensitivities 


Within  each  graph  of  the  prediction  profiler,  the  slope  of  the  response  curve  is  an 
indication  of  the  sensitivity  of  the  system  to  that  factor.  The  user  should  leverage  this 
fact  to  aid  in  the  decision-making  process  for  system  adjustments.  The  assumption  is  that 
implementing  changes  in  factors  requires  time  and  human/financial  resources,  and  that 
time  and  resources  are  limited.  It  follows  that  the  most  efficient  use  of  these  limited 
resources  would  be  to  apply  them  toward  changing  the  more  influential  factors  first. 

C.  DESIRABILITY  FUNCTIONS  (OPTIMIZATION) 

Within  the  JMP  prediction  profiler  platform  resides  an  option  to  set  and 
manipulate  system  desirabilities.  A  desirability  graph  is  displayed  for  each  response  in 
Figure  5.  For  each  response,  JMP  knows  the  range  of  values  that  occur.  When  setting  a 
desirability,  JMP  will  display  a  high,  middle,  and  low  value  for  the  response.  If,  for 
example,  the  average  treatment  duration  were  desired  to  be  minimized,  JMP  would 
recommend  a  0.9819  desirability  for  a  value  of  8.75  weeks  (the  lowest  value  in  the 
experiment  for  average  TBE).  JMP  would  then  recommend  a  0.5  desirability  for  15.125 
weeks  (the  middle  value  for  that  response),  and  a  0.066  desirability  for  21.5  weeks  (the 
highest  value). 

In  this  manner,  the  desirability  of  each  response  was  programmed.  The 
desirabilities  are  summarized  in  Table  10.  For  high,  middle,  and  low,  values  are  on  the 
left  and  the  desirability  for  that  value  is  indicated  on  the  right.  Note  that  the  goal  is  to 
maximize  timeliness  and  capacity,  while  the  specific  goal  of  90%  provider  utilization  is 
desired. 
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Attribute 

Measured  Response 

High  Value 
&  Desirability 

Middle  Value 
&  Desirability 

Low  Value 
&  Desirability 

Efficient 

Provider  Utilization 

0.96 

0.0183 

0.9 

1.0 

0.3 

0.0183 

Capacity 

Throughput 

365 

0.9819 

277.5 

0.5 

190 

0.066 

Timely 

Treatment  Duration 

21.5 

0.066 

15.125 

0.5 

8.75 

0.9819 

Timely 

Time  Between  Encounters 

3.2 

0.066 

2.3 

0.5 

1.4 

0.9819 

Table  10.  Desirability  table  for  system  responses 


D.  FACTORS  OPTIMIZED  FOR  MAXIMUM  DESIRABILITY 
1.  Optimized  Factor  Settings 

Once  all  desirability  values  were  programmed  and  the  profiler  was  instructed  to 
maximize  overall  system  desirability,  the  prediction  profiler  recalculated  the  response 
curves  and  displayed  the  optimized  results,  as  seen  in  Figure  5.  Note  the  overall  system 
desirability  of  88.1%.  The  optimized  factors  can  also  be  seen  in  Figure  5,  but  they  are 
summarized  (displayed  unshaded)  in  Table  11. 


# 

Factor 

Level  1 

Level  2 

Level  3 

1 

Provider  Count 

2 

3 

4 

2 

Provider  Intensity 

Base 

Less 

Least 

3 

CPT  Group  Size 

1 

5 

9 

4 

CM  Group  Size 

1 

5 

9 

5 

Scaffolding  Session  Count 

3 

6 

9 

6 

Cancellation  Rate 

Base 

Less 

- 

7 

Failure  Rate 

Base 

Less 

Least 

8 

Drop  Rate 

Base 

Less 

-- 

Table  11.  Optimal  factors,  as  determined  by  JMP  to  maximize  desirability  functions 
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2. 


Actual  vs.  Expected  Results 


a.  Provider  Count 

A  provider  count  of  two  was  expected,  as  this  is  the  number  of  providers 
that  are  currently  available  to  handle  the  current  patient  load.  Rough  estimates  for 
provider  utilization  for  the  test  site  (Monterey  CBOC)  in  the  base  case  (prior  to  model 
development)  were  60%.  Therefore,  it  would  not  be  logical  to  add  more  providers  when 
the  desired  efficiency  level  (provider  utilization)  is  90%. 

b.  Provider  Intensity 

The  optimal  factor  setting  of  base  case  was  surprising.  Recall  that  the 
base  case  for  provider  intensity  was  the  most  provider  intense  setting.  Further 
investigation  is  required,  but  it  is  hypothesized  that,  since  reducing  provider  count  to  a 
setting  lower  than  two  was  not  an  option,  provider  intensity  was  able  to  stay  high.  A 
higher  provider-patient  ratio  would  seem  to  indicate  higher  system  quality,  so  this  setting 
is  not  undesirable. 


c.  CPT  and  CM  Group  Sizes 

Group  sizes  of  one  for  these  factors  were  surprising.  It  was  expected  that 
higher  group  sizes  would  lead  to  higher  throughput,  and  that  optimal  factors  for  these 
factors  would  be  close  to  nine.  Further  investigation  is  required,  but  it  is  hypothesized 
that  with  large  groups  comes  high  administrative  overhead.  For  example,  if  a  patient 
drops  from  a  group,  another  patient  must  replace  him.  That  person  may  not  be  at  the 
same  level  as  his  predecessor,  which  would  delay  the  group’s  recovery. 

d.  Scaffolding  Session  Count 

Scaffolding  session  count  at  a  low  optimal  setting  was  expected.  Every 
extra  session  is  noted  as  a  cost  to  the  system. 
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e.  Cancellation ,  Failure,  and  Drop  Rates 

Results  for  these  rates  were  also  expected.  The  decision  maker  must 
decide  whether  the  cost  in  time,  human  resources,  and  funding  are  worth  the 
improvement  in  system  performance. 

3.  Optimal  Responses  (Maximized  Desirability) 

The  optimal  responses  (or  maximized  desirability)  can  also  be  seen  in  Figure  5, 
but  they  are  summarized  in  Table  12. 


Attribute 

Measured  Response 

Result 

Units 

Efficient 

Provider  Utilization 

0.88 

-- 

Capacity 

Throughput 

310 

Patients 

Timely 

Treatment  Duration 

10.0 

Weeks 

Timely 

Time  Between  Encounters 

1.58 

Weeks 

Table  12.  Simulation  responses  resulting  from  optimal  factors 

4.  System  Improvements  Gained  Over  Base  Case 

To  demonstrate  the  improvements  that  are  available  to  the  system,  a  comparison 
was  made  between  the  optimal  and  base  factors.  Using  the  JMP  model,  the  base  factors 
were  entered  into  the  prediction  profiler.  The  base  factors,  along  with  the  optimized 
factors,  are  summarized  in  Table  13. 
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# 

Factor 

Base 

Optim 

1 

Provider  Count 

2 

2 

2 

Provider  Intensity 

Base 

Base 

3 

CPT  Group  Size 

9 

1 

4 

CM  Group  Size 

9 

1 

5 

Scaffolding  Session  Count 

4 

3 

6 

Cancellation  Rate 

Base 

Less 

7 

Failure  Rate 

Base 

Least 

8 

Drop  Rate 

Base 

Less 

Table  13.  Comparison  of  base  and  optimal  factors 

With  the  base  factors  set,  the  system  responses  were  noted,  and  summarized  in 
Table  14.  Note  that  considerable  improvements  can  be  obtained.  In  fact,  all  attributes  of 
the  system  are  improved,  and  overall  system  desirability  can  be  improved  by  80%  over 
the  base  case. 


Attribute 

Measured  Response 

Base 

Optim 

Delta 

%  Improve 

Efficient 

Provider  Utilization 

0.61 

0.88 

0.27 

44% 

Capacity 

Throughput  (patients) 

270 

310 

40 

15% 

Timely 

Treatment  Duration  (weeks) 

17.6 

10.0 

-7.6 

43% 

Timely 

Time  Between  Encounters  (weeks) 

2.30 

1.58 

-0.72 

31% 

Overall  System  Desirability 

0.49 

0.88 

0.39 

80% 

Table  14.  System  improvements  gained  using  optimized  factors  over  base  case 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  SECONDARY  RESEARCH  QUESTIONS 

Recall  that,  to  answer  the  primary  research  question,  the  definitions  of  efficiency , 
capacity ,  and  quality  needed  to  be  determined. 

1.  Definition  of  Efficiency  in  PTSD  Health-Care  Delivery 

As  the  system  analysis  has  shown,  PTSD  health-care  delivery  is  a  human 
resource-intensive  endeavor.  Therefore,  efficiency  in  the  system  implies  efficiency  in  the 
use  of  providers.  Accordingly,  the  definition  of  efficiency  in  PTSD  health-care  delivery 
is  high  provider  utilization. 

2.  Definition  of  Capacity  in  PTSD  Health-Care  Delivery 

Defining  capacity  of  the  system  was  intuitively  accomplished.  The  system  should 
treat  as  many  patients  as  possible.  Therefore,  patient  throughput  was  measured  to  assess 
capacity. 


3.  Definition  of  Quality  in  PTSD  Health-Care  Delivery 

Quality  is  a  patient  determination.  For  this  research,  a  patient  is  assumed  to  want 
to  become  as  healthy  as  possible  as  soon  as  possible.  The  latter  refers  to  the  timeliness  of 
the  system.  Therefore,  average  treatment  duration  and  average  time-between-encounters 
were  measured  to  determine  timeliness ,  and  therefore  quality.  In  future  work,  treatment 
efficacy  will  be  incorporated  into  the  model.  In  doing  so,  a  full  measure  of  quality  will  be 
obtained. 

B.  PRIMARY  RESERCH  QUESTION 

Recall  that  the  research  question  was,  “For  the  system  of  PTSD  Health-care 
delivery,  what  are  optimal  factors  that  will  maximize  efficiency,  capacity,  and  qualityV 

Assuming  the  models  are  valid,  the  factors  in  Table  15  should  be  attempted  at  the 
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Monterey  CBOC.  In  real  terms,  this  would  involve  dissolving  group  therapy,  lowering 
the  average  scaffolding  session  count,  implementing  training  and  policies  that  would  cut 
the  cancellation  and  drop  rates  by  half,  and  influencing  the  success  rate  of  therapy 
sessions  to  reduce  their  failure  rate  to  10%. 


# 

Factor 

Optim 

1 

Provider  Count 

2 

2 

Provider  Intensity 

75%  2:1 
25%  1:1 

3 

CPT  Group  Size 

1 

4 

CM  Group  Size 

1 

5 

Scaffolding  Session  Count 

3 

6 

Cancellation  Rate 

9% 

7 

Failure  Rate 

10% 

8 

Drop  Rate 

1% 

Table  15.  Optimized  factors 


If  successful  in  achieving  these  factors,  the  Monterey  CBOC  would  realize  a  44% 
increase  in  efficiency  (provider  utilization),  a  15%  increase  in  capacity  (patient 
throughput),  and  approximately  a  37%  increase  in  quality  (timeliness,  measured  in 
treatment  duration  and  time-between-encounters).  Overall,  the  system  will  achieve  an 
80%  increase  in  desirability. 

C.  OBJECTIVE  ACHIEVEMENT 

Recall  that  the  objective  was  to  provide  the  best  care  in  the  most  efficient  manner 
possible  to  as  many  affected  veterans  as  possible.  The  use  of  the  words  “best,”  “most,” 
and  “to  as  many  as  possible”  categorize  this  objective  as  arguably  unachievable.  The 
mere  act  of  pursuing  the  objective,  however,  will  result  in  improvements  in  the  areas  of 
quality,  efficiency,  and  capacity.  Toward  that  end,  it  has  been  shown  that  improvements 
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can  be  made  in  these  areas,  assuming  the  model  is  valid.  It  is  fair  to  say,  therefore,  that 
this  research  is  a  positive  step  toward  achievement  of  that  objective,  as  discussed  above. 


D.  SUMMARY  OF  RECOMMENDATIONS  FOR  FUTURE  WORK 

As  the  objective  is  a  constantly  moving  target,  there  will  continue  to  be  work  in 
this  field.  Given  the  grave  human  aspect  of  the  problem,  it  should  continue  to  benefit 
from  continued  and  abundant  support.  Suggestions  for  future  work  have  been  mentioned 
throughout  this  thesis.  They  are  summarized  here. 

1.  Validate  the  Models 

Conduct  an  initial  validation  of  the  models  and  repeat  validations  at  each  iteration 
of  major  improvement. 

2.  Improve  the  Measurement  of  Quality 

Incorporate  efficacy  for  the  various  treatment  paths  into  the  model.  Compare  the 
entrance  and  exit  PTSD  scores  from  standardized  psychological  tests  and  incorporate  the 
efficacy  of  treatment  plans  into  the  models. 

3.  Improve  the  Modeling  of  Human  Resources 

Develop  multiple  human  resource  categories  (e.g.,  staff,  multiple  levels  of 
qualification  within  the  provider  ranks).  Leverage  the  ARM  feature  of  ExtendSim  8, 
which  will  prove  to  be  a  valuable  tool  for  this  effort. 

4.  Improve  the  Alignment  of  the  Models  with  Reality 

Enhance  the  models  to  bring  them  to  a  higher  degree  of  alignment  with  reality. 
This  will  cause  the  models  to  become  more  complex,  but  it  will  increase  the  ability  of  the 
decision  maker  to  more  finely  tune  the  real-life  system. 

For  the  probabilistic  design  factors,  a  more  detailed  analysis  of  the  historical  data 
should  be  conducted.  This  may  uncover  smoother  distributions  that  may  result  in  an 

improved  representation  of  reality  within  the  models. 

43 


To  account  for  patient  referrals  that  are  discovered  to  be  in  error  (i.e.,  after 
assessment,  the  patient  is  discovered  to  not  have  PTSD),  incorporate  a  bogus  referral  rate 
into  the  model. 

Investigate  the  characteristics  of  the  medication  management  program  to  develop 
a  more  precise  modeling  of  this  activity.  This  would  require  modifications  to  the  VistA 
medical  information  system,  as  it  is  not  currently  set  up  to  track  this  information. 

Investigate  the  characteristics  of  the  patient  drop  rate,  patient  cancellation  rate, 
and  encounter  failure  rate  to  develop  a  more  precise  modeling  of  these  activities. 
Discovering  categories  for  these  rates,  and  linking  them  to  events  within  the  model  would 
help  significantly  to  discover  the  optimal  application  of  resources  toward  their  reduction. 

5.  Improve  VA  Data  Tracking  and  Presentation 

Gathering  system  statistics  were  a  challenge  for  this  project.  The  VA  information 
system  technician  worked  long  hours  to  gather  information  that  should  (in  a  system 
designed  to  continuously  improve)  be  readily  available.  We  contend  that  future  work 
must  include  a  redesign  of  the  VA  medical  information  system. 
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APPENDIX  A:  EXTENDSIM  SCREEN  SHOTS 


Figure  6  through  Figure  11  display  screen  captures  of  the  ExtendSim  model. 
They  are  inserted  here  to  provide  a  visual  sense  of  the  complexity  of  the  ExtendSim 
model. 

Figure  6  displays  a  view  of  the  entire  model,  including  the  control  section.  Areas 
of  the  model  have  been  blocked  off  and  labeled  to  generally  show  what  sub-functions  are 
being  supported. 

The  rest  of  the  screen  shots  are  exploded  views  of  various  sections  of  the  model. 
Figure  7  shows  the  control  section  of  the  model,  including  the  executive  block  (clock 
icon  that  controls  the  discrete  time  activity  of  the  model),  provider  count  and  resource 
blocks,  provider  utilization  calculators,  provider  shift  schedules,  and  the  scenario 
manager  block.  Figure  8  shows  the  receive,  assess,  and  prepare  activities  as  they  are 
represented  in  model.  Figure  9  shows  how  traditional  therapy  is  represented  in  the 
model,  along  with  the  tracking  activities  as  they  support  the  continuously  improve 
function.  Figure  10  shows  how  medication  management  is  represented  in  the  model. 
Figure  1 1  shows  an  exploded  view  of  the  scaffolding  block,  which  supports  the  prepare 
patient  function. 
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Figure  6. 


Extend  Sim  Model  of  PTSD  System  (Entire  View) 
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Figure  7.  Control  section  of  ExtendSim  model 


Figure  9.  Traditional  therapy  represented  in  ExtendSim  model 
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Figure  1 1 .  Detail  of  “Prepare  Patient”  block  seen  in  Figure  6. 
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APPENDIX  B:  JMP  MODEL  FIT  STATISTICS 


Table  16  shows  that  most  factors  correlate  with  the  responses  (i.e.,  probability  < 
0.0001  of  t  ratio  assuming  null  hypothesis  is  true).  Figure  12,  Figure  13,  Figure  14,  and 
Figure  15  show,  for  each  response,  plots  of  how  well  the  JMP  model  was  able  to  predict 
results  from  the  ExtendSim  simulation.  Actual  ExtendSim  simulation  results  are  shown 
as  gray  dots.  The  JMP  prediction  is  indicated  by  a  solid  line. 

It  can  be  seen  at  a  glance  that  this  model  fits  well.  The  horizontal  dashed  line 
(sample  mean  of  the  response)  falls  well  outside  the  bounds  of  the  95%  confidence 
curves  (dotted  lines  surrounding  solid  line  of  JMP  prediction),  indicating  the  model  is 
significant.  The  response  p-values  (all  are  below  0.0001),  R2,  and  root  mean  square  error 
(RMSE)  appear  below  the  plot.  The  RMSE  is  an  estimate  of  the  standard  deviation  of  the 
system  noise,  assuming  that  the  unestimated  effects  are  negligible.  Note  that  quantity 
complete  has  a  large  deviation,  but  all  other  responses  are  tightly  correlated. 
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Resp. 

Factor 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

9.950346294 

0.036895858 

269.6S*3549 

<0.0001 

Provider  C  ount 

-0.094277SS1 

0.009003901 

-10.4*078162 

<0.0001 

C'PT  Group  Size 

0.30613S024 

0.002250975 

136.002395  3 

<0.0001 

- 

CM  Group  Size 

0.406729255 

0.0022509*5 

180.6902395 

<0.0001 

Scaffolding  Session  Count 

0.236315734 

0.0030013 

7873778314 

<0.0001 

5 

Cancellation  RateTBase  Casel 

0.43828*439 

0.007351654 

59  61*52523 

<0.0001 

j 

Success  RatefBase  C'asel 

1.064221542 

0.010396809 

102.3603985 

<0.0001 

4> 

y 

Success  RatefHigherl 

-0.088334096 

0.010396809 

-8.496269*29 

<0.0001 

Drop  RatefBase  Casel 

-0.21 50SS6S5 

0.00*351654 

-29.25*18144 

<0.0001 

2 

Provider  Intensity  [Base  C'asel 

0.06029373 

0.010396809 

5799253241 

<0.0001 

Provider  Intensity  [Lessl 

0.00*04810' 

0.010396809 

0.6**910581 

0.4978 

Intercept 

0  59936608 

0.003277707 

414.9048826 

<0.0001 

Prosider  Count 

-0.013079944 

0.000T99S"* 

-16  3524443* 

<0.0001 

CPT  Group  Size 

0.05068636* 

0.000199969 

253.470S03S 

<0.0001 

CM  Group  Size 

0.052350811 

0.000199969 

261794301* 

<0.0001 

— 

S 

Scaffolding  Session  Count 

0.043*1071 

0.000266626 

163.940364* 

<0.0001 

= 

= 

Cancellation  RatefBase  Casel 

0.0695809 

0.00065309* 

106.5399419 

<0.0001 

w 

a 

Success  RatefBase  Casel 

-0.090431302 

0.000923618 

-9*  90980*56 

<0.0001 

y 

r 

- 

Success  RatefHigherl 

-0.00101*726 

0.00092361S 

-1.10 1 S9044S 

07705 

4) 

Drop  RatefBase  Casel 

0.018911024 

0.00065309* 

28.95592558 

<0.0001 

Provider  IntensitvfBase  Casel 

0.006211004 

0.000923618 

6724642*S6 

<0.0001 

Provider  IntensitvfLessl 

0.000383983 

0000923618 

0.415*3*295 

0.67*6 

Intercept 

294.3S4430* 

0.923462545 

31S7S32925 

<0.0001 

Provider  Count 

-0.101508916 

0.225357691 

-0.45043466* 

0.6524 

-> 

CPT  Group  Size 

-0.34400*202 

0.056339423 

-6.1059*6695 

<0.0001 

CM  Group  Size 

-0.5109310* 

0.056339423 

-9.06SS02022 

<0.0001 

Scaffolding  Session  Count 

-0.307613169 

0.075 1 1923 

-4  09499895 

<0.0001 

Cancellation  RateTBase  Casel 

-2.469364426 

0.1S4003*S4 

-13.42018286 

<0.0001 

Success  RatefBase  C'asel 

-5.796296296 

0.26022064* 

-2277454421 

<0.0001 

Success  RatefHigherl 

0.500342936 

026022064* 

1  9227641* 

0.0545 

> 

Drop  RatefBase  Casel 

-1 0.800 1 829 

0.1S4003*S4 

-58.69543916 

<0.0001 

Provider  IntensitvfBase  Casel 

-0.184499314 

0.26022064* 

-0709011051 

0.4783 

Provider  IntensitvfLessl 

-0.06*901235 

0.26022064* 

-0.26093*153 

07941 

Intercept 

1.108718915 

0.00259289* 

427.5984694 

<0.0001 

Pro  vider  Count 

-0.10*813982 

0.000632*59 

-1  *0.387085 

<0.0001 

CPT  Group  Size 

-0.02316515 

0.00015819 

-146.4389**2 

<0.0001 

CM  Group  Size 

-0.01499369* 

0.00015819 

-947S296992 

<0.0001 

3 

Scaffolding  Session  Count 

0.006826463 

0.00021092 

32.36522229 

<0.0001 

Cancellation  RatefBase  Casel 

0.00200763* 

0.000516646 

3.8859070*1 

0.0001 

3 

Success  RatefBase  Casel 

0.013122735 

0.000*3064* 

1*. 96042422 

<0.0001 

Success  RatefHigherl 

-0.00029207S 

0.000*3064* 

-0.39975255 

0.6893 

Drop  RatefBase  Casel 

-0.005241223 

0.000516646 

-10.144*1496 

<0.0001 

Provider  IntensitvfBase  Casel 

0.022*9S066 

0.000*3064* 

31.20256031 

<0.0001 

Provider  IntensitvfLessl 

0.00071393 

0.000*3064* 

0.9*7119784 

0.3285 

Table  16.  JMP  model  fit  statistics 
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Actual  by  Predicted  Plot 


Figure  12.  Actual  provider  utilization  responses  predicted  by  JMP  model 


Actual  by  Predicted  Plot 


Quantity  Complete 
P<?0001" RSq=0.33 


Figure  13.  Actual  quantity  complete  responses  predicted  by  JMP  model 
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Figure  14. 


Figure  15. 


Actual  by  Predicted  Plot 


Actual  average  treatment  duration  responses  predicted  by  JMP  model 


Actual  by  Predicted  Plot 


Average  TBE 
P<0001  RSq=0.95 


Actual  average  time-between-encounters  responses  predicted  by  JMP  model 
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